Incremental Computation of Linear Machine Learning Models in Parallel Database Systems

We study the serial and parallel computation of Γ (Gamma), a comprehensive data summarization matrix for linear machine learning models widely used in big data analytics. We prove that computing Gamma can be reduced to a single matrix multiplication with the data set, where such multiplication can be evaluated as a sum of vector outer products, which enables incremental and parallel computation, essential features for scalable computation. By exploiting Gamma, iterative algorithms are changed to work in two phases: (1) Incremental-parallel data set summarization (i.e. in one scan and distributive); (2) Iteration in main memory exploiting the summarization matrix in intermediate matrix computations (i.e. reducing number of scans). Assuming the machine learning model is based on Gaussian distributions, we show that the covariance (and correlation) matrix, present in every Gaussian model, can be derived directly from Gamma. Therefore, many intermediate computations on large matrices collapse to computations based on Gamma, a much smaller matrix. We justify it is necessary to develop specialized database algorithms for dense and sparse matrices, respectively, and we introduce a density threshold to decide either algorithm. Assuming a distributed memory model (i.e. shared-nothing) and a larger number of points than processing nodes, we show computing Gamma exhibits close to linear speedup. We study how to compute Gamma with existing database systems processing mechanisms and their impact on time complexity. At the same time we also highlight weaknesses and limitations of our proposal.